Minimum risk methods in the estimation of unknown sparsity

The selection of significant components in a sparse vector, directly observed with i.i.d. observational noise, typically proceeds by thresholding the observations. The objective in this paper is to choose the threshold that minimizes the risk (expected squared prediction error) of the estimator with respect to the noise-free sparse vector. The risk as a function of the model size (or, equivalently, as a function of the threshold value) is estimated by Stein’s Unbiased Risk Estimator (SURE) and by Generalized Cross Validation (GCV). Minimization of these criteria leads to additional randomness, generally not taken into account. A correction for this minimization randomness is proposed. The analysis starts from the observation that soft-thresholding data with normal noise allows unbiased risk estimators. The step towards hardthresholding is then generalized to other noise models and — through a link between SURE and Mallows’ Cp — to other selection criteria. The case of Akaike’s Information Criterion (AIC) is further elaborated. In a second theme of this paper, the use of GCV for sparse model selection is motivated by two new theorems, one for the asymptotic behavior near the optimal threshold, the other for the asymptotic behavior for thresholds close to zero. A third theme of this paper is the application of GCV to iterative threshold procedures for estimating sparse vectors from indirect observations, such as in inverse problems, compressed sensing and so on. We propose to iterate with a step-dependent threshold and find experimentally that this allows closer fit than any fixed threshold. keywords sparsity, thresholding, wavelet, model selection, Mallows’ Cp, Stein Unbiased Risk Estimation, generalized cross validation, robust AMS 2000 subject classification Primary 62J07,62J02; secondary 62G08